Beijing Zhiyuan Artificial Intelligence Research Institute announces the launch of the native multimodal world model Emu3. This model is based on next-token prediction technology and does not rely on diffusion models or combinatorial methods to achieve understanding and generation across text, image, and video modalities. Emu3 surpasses existing well-known open-source models such as SDXL, LLaVA, and OpenSora in tasks like image generation, video generation, and visual language understanding, showcasing exceptional performance.

Apple Inc. recently launched a significant update for its multimodal AI model MM1, upgrading it to version MM1.5. This upgrade represents more than just a simple change in version number; it is a comprehensive enhancement of capabilities, showcasing the model's stronger performance across various fields. The core upgrade of MM1.5 lies in its innovative data processing methods. The model utilizes a data-centric training approach, carefully selecting and optimizing the training dataset. Specifically, MM1.5 employs high-definition OCR data and synthetic image descriptions, along with optimizations.

Recently, Japan-based startup Rhymes AI unveiled their first artificial intelligence model — Aria. The company claims that Aria is the world's first open-source multimodal mixture of experts (MoE) model. This model not only possesses the capability to handle various input modalities but also claims to be on par with, or even superior to, several well-known commercial models. The design philosophy of Aria aims to provide exceptional understanding and processing capabilities across diverse input forms including text, code, images, and videos.

Recently, Apple's AI research team launched their next-generation family of Multimodal Large Language Models (MLLMs) - MM1.5. This series of models can integrate various data types such as text and images, showcasing new capabilities of AI in understanding complex tasks. Tasks like visual question answering, image generation, and multimodal data interpretation can be better addressed with the help of these models. A major challenge for multimodal models is how to achieve effective interaction between different data types. Previous models often struggled in this aspect.

Meta has recently confirmed that photos and videos taken by users using Ray-Ban Meta smart glasses may be used to train its AI models. This practice has raised concerns as users may not be aware that they are providing Meta with a substantial amount of images for training its AI models, which could include their home interiors, family members, or personal documents. Emil Vazquez, Meta's policy communications manager, stated that in regions where multimodal AI is available, ...

Recently, the Allen Institute for AI (Ai2) released Molmo, a brand new open source multimodal AI model family that performs exceptionally well, even surpassing OpenAI's GPT-4o, Anthropic's Claude 3.5, and Google's Gemini 1.5 in several third-party benchmark tests. Molmo not only accepts user-uploaded images for analysis but has also been trained using '1000 times less data than its competitors.'

Recently, an open source multimodal AI model named Molmo has drawn widespread attention in the industry. This AI system, based on Qwen2-72B and leveraging OpenAI's CLIP as the visual processing engine, is challenging the dominance of traditional commercial models with its outstanding performance and innovative features. Molmo's standout characteristic is its efficient performance. Despite its relatively small size, it can compete with competitors that are ten times larger in processing capability. This 'small but exquisite' design philosophy not only enhances the model's

Recently, edge computing chip and software startup SiMa.ai announced the launch of its new product—MLSoC Modalix chip, further solidifying its position in the edge AI market. SiMa.ai has just completed a $70 million funding round, with investors including industry giants like Dell Technologies Capital.

Salesforce has released an open source multimodal AI model named xGen-MM, designed to simultaneously understand and generate various data types such as text and images, significantly changing the way AI research and applications are conducted. The model performs exceptionally well in multiple benchmark tests, demonstrating strong performance compared to similar open source models, and includes pretrained models, datasets, and fine-tuning code. The largest model boasts 4 billion parameters and can handle 'interleaved data,' enabling multitasking, such as answering questions about multiple images at once. The diverse options available in the model reflect the potential of AI.

Domestic large models have shown strong capabilities in the multimodal field of artificial intelligence. Tencent's Hunyuan-vision and Shanghai AI Lab's InternVL2-40B are leading their respective closed-source and open-source fields, surpassing internationally renowned models. Hunyuan-vision, as the multimodal version of Tencent's Hunyuan large model, is favored by developers for its API calls and is offered for free in Tencent's Yuanbao APP, showcasing practicality and ease of use. Yuanbao APP ranks first in China for multimodal capability assessments through a series of tests.

The 2nd MER24 Challenge focused on the application of AI in emotional recognition within real human-computer interaction scenarios, attracting renowned international scholars. The voice technology team from Soul App excelled in the highly challenging Semi track, winning first place. Their success stemmed from innovations in multimodal data understanding, emotional recognition algorithms, model optimization, and efficient collaboration. To tackle ....

Meta has stated that it will not launch its upcoming multimodal AI model in the EU due to regulatory concerns, which will prevent European companies from using the model, despite it being released under an open license. Meta spokesperson Kate McLaughlin told The Verge, "In the coming months, we will release the multimodal Camel model, but we will not launch it in the EU due to the unpredictability of the European regulatory environment."The EU has recently set a compliance deadline for the new A

Meta announced to Axios on Wednesday that due to the unclear regulations in the EU, the company will not release its new multimodal AI models or future versions to customers in EU countries. This decision has sparked a broad discussion on the AI regulatory environment in the EU and reflects the strategic adjustments of American tech giants in the face of global regulatory requirements.Meta's multimodal AI models have the ability to process video, audio, images, and text and were originally plann

2023 has been one of the most disruptive years in the field of artificial intelligence, with a plethora of generative AI products entering the mainstream. Continuing its transformative journey, generative AI is expected to transition from an exciting topic to real-world applications in 2024. The field of generative AI is rapidly evolving, giving rise to a series of widespread trends that will facilitate the adoption of AI across various industries and its presence in our daily lives. Generative AI models are transcending text creation through the integration of multimodal functionalities.

Unified-IO2 marks a significant breakthrough in the field of artificial intelligence, featuring autoregressive capabilities that can handle various data types including text, images, audio, and video. The innovative single encoder-decoder transformer model overcomes the limitations of previous models in multimodal data processing. It excels in performance, setting new records in GRIT assessments across 35 datasets, particularly surpassing competitors in image generation. Unified-IO2 employs complex and innovative methods, including shared representation spaces and pretrained visual transformers.

The three most important innovations in the field of artificial intelligence in 2023 are Multimodal AI, Constitutional AI, and Text-to-Video Technology. Multimodal AI refers to the ability of AI to process various types of data, including text, images, videos, and audio. Constitutional AI aims to align AI with human values by drafting a constitution of values and employing reinforcement learning methods to control AI behavior. Text-to-Video Technology is a significant achievement with investments of billions of dollars, allowing the generation of videos based on text and changing video styles. These innovations will have a profound impact on the field of AI and alter the way people interact with technology.

Meta has recently announced the launch of multimodal AI features for Ray-Ban smart glasses. Utilizing the glasses' camera and microphone, users can experience audiovisual perception of their surroundings. Features include clothing coordination suggestions, text translation, image descriptions, and various other applications. Beta testing will take place in the United States, covering AI functionalities such as photo tagging, translation, and summarization.

🤩 Progress in Multimodal AI: The AI video generation tool Pika is at the forefront of the industry. Multimodal AI is moving towards scenario-based, practical, and commercial applications. 💡 Competition in the Multimodal AI Market: Companies like Runway, Stability AI, and Midjourney are also entering the AI video generation arena. 🔍 Future of Multimodal AI: Large multimodal models are expected to bridge the gap between the physical and digital worlds.

AI startup Reka has launched its multimodal AI assistant Yasa-1 to compete with OpenAI's ChatGPT. Yasa-1 supports 20 languages, capable of processing large documents and executing code, outpacing Claude2 in speed. What sets Yasa-1 apart is its multimodal capability, enabling it to integrate text with multimedia files and perform programming tasks. Reka plans to expand the usage of Yasa-1 and enhance its features, marking an intensifying competition in the multimodal AI assistant space.

OpenAI is preparing to launch the multimodal AI system GPT-Vision to compete with Google Gemini. OpenAI is also developing a more powerful multimodal model codenamed Gobi. The multimodal AI is entering a phase of practical application, leading to direct competition between OpenAI and Google.